Note: This is intended to possibly be extracted into a library, so the
docs are written as if this were its own library. Cargo specific code
(besides the use of `CargoResult`) will go in its own module for the
same reason.

This adds an MVP background queueing system intended to be used in place
of the "try to commit 20 times" loop in `git.rs`. This is a fairly
simple queue, that is intended to be "the easiest thing that fits our
needs, with the least operational impact".

There's a few reasons I've opted to go with our own queuing system here,
rather than an existing solution like Faktory or Beanstalkd.

- We'd have to write the majority of this code ourselves no matter
  what.
  - Client libraries for beanstalkd don't deal with the actual job
    logic, only `storage.rs` would get replaced
  - The only client for faktory that exists made some really odd API
    choices. Faktory also hasn't seen a lot of use in the wild yet.
- I want to limit the number of services we have to manage. We have
  extremely limited ops bandwidth today, and every new part of the stack
  we have to manage is a huge cost. Right now we only have our server
  and PG. I'd like to keep it that way for as long as possible.

This system takes advantage of the `SKIP LOCKED` feature in PostgreSQL
9.5 to handle all of the hard stuff for us. We use PG's row locking to
treat a row as "currently being processed", which means we don't have to
worry about returning it to the queue if the power goes out on one of
our workers.

This queue is intended only for jobs with "at least once" semantics.
That means the entire job has to be idempotent. If the entire job
completes successfully, but the power goes out before we commit the
transaction, we will run the whole thing again.

The code today also makes a few additional assumptions based on our
current needs. We expect all jobs to complete successfully the first
time, and the most likely reason a job would fail is due to an incident
happening at GitHub, hence the extremely high retry timeout.

I'm also assuming that all jobs will eventually complete, and that any
job failing N (likely 5) times is an event that should page whoever is
on call. (Paging is not part of this PR).

Finally, it's unlikely that this queue will be appropriate for high
thoughput use cases, since it requires one PG connection per worker (a
real connection, adding pg bouncer wouldn't help here). Right now our
only background work that happens is something that comes in on average
every 5 minutes, but if we start moving more code to be run here we may
want to revisit this in the future.

The `Runner` is basically the main interface into the job queue, and
stores the environment, thread pool, and database connection pool. We
technically don't need a thread pool since our jobs are so infrequent
that we don't need more than one thread. But our tests are simplified by
having it, and a general purpose lib will need it.

The other oddity is that we will want to panic instead of returning a
result if updating or fetching a job failed for some reason. This isn't
something we expect to occur unless there's an issue in the DB, so this
seems fine. A thread panicking won't kill the runner. We are also only
panicking if changes to the `background_jobs` table fails, not if the
job itself fails.

This fundamentally changes the workflow for all operations we perform
involving git, so that they are not performed on the web server and do
not block the response. This will improve the response times of `cargo
publish`, and make the publish process more resilient, reducing the
liklihood of an inconsistency occurring such as the index getting
updated, but not our database.

Previously, our workflow looked something like this:

- When the server boots, do a full clone of the index into a known
  location
- Some request comes in that needs to update the index
- Database transaction is opened
- Local checkout is modified, we attempt to commit & push (note: This
  involves a mutex to avoid contention with another request to update
  the index on the same server)
- If push fails, we fetch, `reset --hard`, and try again up to 20 times
- Database transaction is committed
- We send a successful response

The reason for the retry logic is that we have more than one web server,
meaning no server can be sure that its local checkout is actually up to
date. There's also a major opportunity for an inconsistent state to be
reached here. If the power goes out, the server is restarted, something
crashes, etc, in between the index being updated and the database
transaction being committed, we will never retry it.

The new workflow looks like this:

- Some request comes in that needs to update the index
- A job is queued in the database to update the index at some point in
  the future.
- We send a successful response
- A separate process pulls the job out of the database
- A full clone of the index is performed into a temporary directory
- The new checkout is modified, committed, and pushed
- If push succeeds, job is removed from database
- If push fails, job is marked as failed and will be retried at some
  point in the future

While a background worker can be spread across multiple machines and/or
threads, we will be able to avoid the race conditions that were
previously possible by ensuring that we only have one worker with one
thread that handles index updates. Right now that's easy since index
updates are the only background job we have, but as we add more we will
need to add support for multiple queues to account for this.

I've opted to do a fresh checkout in every job, rather than relying on
some state that was setup when the machine booted. This is mostly for
simplicity's sake. It also means that if we need to scale to multiple
threads/processes for other jobs, we can punt the multi-queue
enhancement for a while if we wish. However, it does mean the job will
take a bit longer to run. If this turns out to be a problem, it's easy
to address.

This should eliminate the opportunity for the index to enter an
inconsistent state from our database -- or at least they should become
eventually consistent. If the power goes out before the job is committed
as done, it is assumed the job failed and it will be retried. The job
itself is idempotent, so even if the power goes out after the index is
updated, the retry should succeed.

One other side effect of this change is that when `cargo publish`
returns with an exit status of 0, that does not mean that your crate/new
version is immediately available for use -- if you try to point to it in
Cargo.toml seconds after publishing, you may get an error that it could
not find that version. This was technically already true, since neither
S3 nor GitHub guarantee that uploads/pushes are immediately visible.
However, this does increase the timescale beyond the delay we would have
seen there. In most cases it should be under 10 seconds, and at most a
minute.

One enhancement that will come as a followup, but is not included in
this PR is a UI to see the status of your upload. This is definitely
nice to have, but is not something I think is necessary for this feature
to land. The time it would take to navigate to that UI is going to be
longer than the time it takes the background job to run in most cases.
That enhancement is something I think can go hand in hand with rust-lang#1503
(which incidentally becomes much easier to implement with this PR, since
a "staging" publish just skips queuing the background job, and the only
thing the button to full publish needs to do is queue the job).

This setup does assume that all background jobs *must* eventually
succeed. If any job fails, the index is in an inconsistent state with
our database, and we are having an outage of some kind. Due to the
nature of our background jobs, this likely means that GitHub is down, or
there is a bug in our code. Either way, we page whoever is on-call,
since it means publishing is broken. Since publishing crates is such an
infrequent event, I've set the thresholds to be extremely low.

These binaries are pretty simple (and a bit spaghetti) for the time
being, since we have so little actually using the background job
framework. The runner is meant to be run on a worker dyno continuously,
while the monitor should be run by a cron-like tool roughly every 5
minutes.

We should move `update-downloads` to be scheduled as well, since it
spends so little time actually working.

Now that we're wrapping the connection in our own smart pointer that
doesn't implement `Connection` we need some explicit derefs (adding the
manual `Connection` impl isn't worth avoiding these `*`s). We need to be
able to clone the connection pool (only in tests, but this also only
requires modifying the test variant), so we need the `Arc`.

Similarly, since in tests the connection is a re-entrant mutex, we can't
grab the connection before spawning the worker thread. The lock isn't
`Send` that's for a very good reason. So we instead need to clone a
handle to the pool and grab the connection on the thread we intend to
use it.

Previously, we'd immediately update the database and then queue the
index update to be run later. Jobs aren't guaranteed to run in the
order they were queued (though in practice they likely will). This means
that if someone yanked and unyanked a crate in rapid succession, the
database may end up permanently out of sync with the index.

With this change, the final result may still be out of order, but the
database will be in sync with the index whatever the outcome is. Since
rapidly yanking and unyanking isn't something we expect users to do, and
even if they did jobs should be running in order under normal
circumstances, I don't think we need to do anything to ensure more
consistency than this.

When testing in staging, the runner started crashing on boot. This was
because we only set the connection pool size to 1 when we meant to set
it to 2 (one for the runner, one for the worker thread). So in each
loop, if all jobs took greater than 1 second to run, the runner would
crash.

This fixes the pool size, and also does not return an error if no
database connection could be retrieved from the pool.

Right now we check if a crate is already yanked/unyanked before enqueing
the job at all. This means that if you try to undo a yank before the
yank finishes running, we won't enqueue the unyank at all. This isn't an
expected scenario, and we still might not do the unyank if the jobs are
run out of order, but this means that we should always end up in the
expected state under normal operation.

The time a full clone takes caused too long of a delay when publishing a
crate. This instead performs a full clone when the runner boots, and
then locks and does `git fetch && git reset --hard origin/master` at the
top of every job. The reset means we can (somewhat) scale to multiple
runners, and retains unwind safety in the environment.

Note that there can still be a delay measured in minutes if a publish is
performed right after the server boots, as the job runner starting up
does not block the web server from starting.